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Executive summary 


The project 


Promoting Alternative Thinking Strategies (PATHS) is a school-based social and emotional learning 
(SEL) curriculum that aims to help children in primary school manage their behaviour, understand their 
emotions, and work well with others. PATHS consists of a series of lessons that cover topics such as 
identifying and labelling feelings, controlling impulses, reducing stress, and understanding other 
people’s perspectives. It is delivered twice weekly in 30-40 minute lessons by teachers to all children 
in a given class, typically in the slots allocated for Personal, Social and Health Education. This 
curriculum is supplemented by activities that support the application of new skills during the school day 
and activities that are sent home to parents that cover the topics taught in class. 


In this trial, 45 participating schools from Greater Manchester were randomly allocated to implement 
PATHS for two years (school years 2012-2013 and 2013-2014) or continue their usual practice during 
the same period. This evaluation focuses on the academic outcomes of the intervention for children in 
Years 5 and 6 in English/reading and maths. 


The main trial was funded by the National Institute for Health Research (NIHR) and focused on the 
impact of PATHS on the social-emotional wellbeing of children in Years 3-5. The outcomes of this study 
include: social and emotional competence, quality of life, and attendance. The Education Endowment 
Foundation (EEF) provided additional funding for PATHS training and materials, in addition to the 
evaluation of the academic outcomes detailed in this report. PATHS materials were provided by 
Barnardo’s, and training was led by staff from the Evidence-Based Prevention and Intervention Support 
Centre at Pennsylvania State University (PSU). Teachers implementing PATHS were supported by 
trained coaches. The findings of the NIHR trial will be published at a later date. 


Key Conclusions 


1. Overall, PATHS did not have a positive impact upon children’s academic attainment 


2. The evidence does not indicate that PATHS produced differential gains for children eligible 
for free school meals (FSM). 


academic outcomes 
average, only half of the lessons were delivered. 


5. While PATHS does not appear to be a cost-effective way of improving students’ attainment, 


it may have an impact on other, non-cognitive outcomes. A full report on the NIHR trial 
looking at these outcomes will be published at a later date. 


Security rating 


The findings for Year 6 may be considered of high security, while the Year 5 findings are considered to 
have moderate to high security. The evaluation was set up as a randomised controlled trial that aimed 
to compare the progress of pupils who received the programme to very similar pupils who did not. The 
trial was an efficacy trial, which tested the project in schools where support was made available to 
optimise implementation conditions (e.g. training led by programme developers, coaching support for 
teachers). 


There were 1,705 Year 5 children and 1,631 Year 6 children in the 45 participating schools. The number 
of schools that dropped out of the project was less than 20% for the Year 5 cohort, and 0% for the Year 
6 cohort. The low dropout rate for Year 6 increases the security of the results and the accuracy of the 
estimate of the effect it was possible to obtain. In the case of the negative finding in Year 6 English, it 
appears that the result is attributable to the intervention, rather than chance. 
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Findings 


e Our analyses indicated that PATHS did not have a positive impact on academic outcomes. The 
effect sizes produced in the study were mixed and extremely modest, with several analyses 
favouring students that did not take part in the intervention. 

e Teachers felt that there was a lack of time to implement PATHS at the recommended frequency. 
This meant that on average only half of the lessons were delivered. The delivery context of the 
trial was representative of real-world conditions, and any attempt to implement PATHS ona 
wider scale would likely face similar challenges. 

e The implementation and process evaluation showed that the impact of PATHS was greater 
when delivered with higher levels of implementation quality, although these effects could be the 
result of better teaching overall, rather than the programme itself. 

e Similarly, higher levels of reach were associated with better academic outcomes, although this 
effect may reflect the influence of attendance patterns in participating schools. 

e This was the first study in the UK to focus on the impact of PATHS on academic attainment. 
Previous UK studies have focused on the outcomes of children’s social-emotional competence, 
behaviour, and/or mental health rather than academic attainment. 

e Aprevious randomised controlled trial from the USA showed evidence that PATHS can improve 
aspects of children’s academic attainment, albeit to variable and modest degrees. However, 
the American study experienced an attrition rate of nearly 50%, raising questions regarding the 
security of the analyses. 

e Recent evidence, including the evidence in the Teaching and Learning Toolkit, suggests that 
high quality, well implemented SEL programmes can impact positively upon a range of 
outcomes for children, including their academic attainment. 

e Given the quality of the evaluation, it is likely that the lack of impact upon children’s academic 
attainment in this trial can be attributed to the lack of success of the PATHS programme itself. 

e The mixed findings may indicate issues in relation to the cultural transferability of the 
programme. Concerns about whether the programme had been transferred successfully to a 
UK context were raised by teachers that took part in the trial 


Effect size Estimated Security rating 
months’ progress 
Year 5 Maths vs. rl 
contol 0.026 1 month 4888 £ 
FSM vs. control -0.036 -1 month - - 
Year 5 Reading ; : 
Se. sani 0.029 1 month 42888 £ 
FSM vs. control 0.017 0 months - - 
Yeare Mats ve: -0.025 -1 month 88888 £ 
control ; 
FSM vs. control 0.016 0 months — — 
Year 6 English 
econ -0.106 -2 months 88888 £ 
FSM vs. control 0.120 +2 months — - 
Cost 


For a single form entry school, the initial cost of PATHS is £6,532. This figure includes the essentials 
for PATHS implementation: a curriculum pack for each year group from Reception to Year 6, 
additional/supplementary materials (for example books, CDs, puppets, ‘feeling faces’, and posters), 
initial training for seven staff, headteacher and/or PATHS co-ordinator training, and two days of 
coaching time. Schools can also choose to purchase only the PATHS curriculum packs at an initial cost 
of £2,100. The average cost per pupil for the full intervention is approximately £11.52 a year, over a 
three year period. 


1 Since this report was published, the conversion from effect size into months of additional progress has been slightly revised. If 
these results were reported using the new conversion, all measures would be reported as 0 months of additional progress rather 
than -1. The Year 6 English results would remain at -2 and +2 months. See here for more details. 
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Introduction 


Intervention 


Promoting Alternative Thinking Strategies (PATHS) is a school-based social and emotional learning 
(SEL) curriculum that helps children to manage their behaviour, understand their emotions, and work 
well with others. It is a universal intervention for all children in a given class. PATHS consists of a series 
of lessons that cover topics such as identifying and labelling feelings, controlling impulses, reducing 
stress, and understanding other people’s perspectives. This SEL curriculum is supplemented by 
activities that support the application of new skills during the course of the school day (for example 
‘teachable moments’ such as using a playground incident to demonstrate the importance of resolving 
conflicts peacefully), and parental materials—such as send-home activities— that aim to extend 
learning to the home environment. A detailed description of PATHS can be found in the Methods section 
of this report. The developer of the programme is based at the Evidence-Based Prevention and 
Intervention Support Centre at Pennsylvania State University (PSU), and has been involved in training, 
promoting, and supporting the programme worldwide for many years. 


PATHS training was led by staff from PSU. The PATHS materials themselves were provided by 
Barnardo’s, which holds the license for the Anglicised version of the curriculum. Ongoing support and 
assistance with implementation was provided by three members of the evaluation team who were 
themselves trained in PATHS by PSU staff and were supervised throughout the trial. 


Additional information can be found at: 
http://www.channing-bete.com/prevention-programs/paths/paths.html 
Background evidence 


Learning is a social process and therefore the extent to which children are able to manage their 
behaviour, understand their emotions, and work well with others will likely influence how well they do in 
school academically. Additionally, the creation of a safe, caring and participatory classroom 
environment is assumed to influence children’s engagement and their readiness to learn. Thus, 
improvements in academic achievement are viewed as corollaries of SEL interventions (‘distal 
outcomes’, see Humphrey, 2013). Three recent meta-analyses have provided evidence to support this 
assertion, demonstrating that high quality, well implemented SEL programmes can impact positively 
upon a range of outcomes for children, including their academic attainment (Durlak, Weissberg, 
Dymnicki, Taylor, and Schellinger, 2011; Sklad, Diekstra, De Ritter, Ben, and Gravesteijn, 2012; 
Wigelsworth, Lendrum, and Oldfield, in press). These analyses report average effect sizes of SEL on 
attainment in the 0.27 (Durlak et al., 2011) to 0.46 (Sklad et al., 2012) range. These impressive findings 
do not appear to be related to methodological features (Such as the use of randomised designs or the 
source of data—Durlak et al, 2011). There is also evidence from individual studies and meta-analyses 
that children from socio-economically disadvantaged backgrounds may experience differential gains 
from exposure to SEL interventions when compared to their more affluent peers, albeit in relation to 
social, emotional and behavioural outcomes rather than academic attainment per se (Holsen et al., 
2009; Wilson and Lipsey, 2007). It should be noted, however, that most studies in this body of evidence 
originate in the United States, and cultural transferability cannot be assumed. Furthermore, around two- 
thirds were led by, or involved, the intervention developers, and so there is a clear need for independent 
replication (Wigelsworth, Lendrum and Oldfield, in press). 


The PATHS curriculum is supported by an extensive international literature base. This includes multiple 
randomised controlled trials (RCTs) (for example Domitrovich, Cortes, and Greenberg, 2007). It has 
been designated as a ‘model program’ by both the Center for Study and Prevention of Violence, USA 
(CSPV, 2006) and the Substance Abuse and Mental Health Services Administration, USA (SAMHSA, 
2011) as a result of clear evidence of its efficacy, sustained effects, and multiple-site replications. 
However, the overwhelming majority of PATHS studies focus on social, emotional and behavioural 
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outcomes. To date there has been limited evidence of the effects of the programme on children’s 
academic attainment. A recently published RCT has indicated that PATHS can improve aspects of 
children’s educational outcomes, albeit to modest and variable degrees: for example, children in PATHS 
schools were (according to Schonfeld et al., 2014) between 1.51 and 1.91 times more likely to achieve 
basic proficiency in reading, writing or maths than those in control schools in some year groups. This 
particular study also demonstrated a natural variation dose-response relationship in intervention 
schools, with more children achieving basic proficiency in reading and maths in classes where PATHS 
was taught more frequently. However, the trial in question experienced an attrition rate of nearly 50%, 
raising questions about the security of the analyses. Two earlier small-scale RCTs of PATHS—one in 
mainstream education and another focusing on deaf students in special education—failed to find 
intervention effects on children’s reading or maths, although a marginal, non-significant trend was found 
in relation to the reading scores of deaf students (Greenberg, Kusche and Riggs, 2004). 


There have been only two RCTs of PATHS in the UK education context to date, yielding mixed findings. 
In both cases, assessment of outcomes focused on children’s social-emotional competence, behaviour, 
and/or mental health. Ross et al.'s (2011) study of a Northern Irish cultural adaptation of the programme 
found that PATHS (rebranded as ‘Together 4 All’) produced effects that were ‘weak and inconsistent, 
but generally in a positive direction’ (p.61). Little et al.’s (2012) trial of PATHS in Birmingham, England, 
yielded null results. The study reported herein is the only UK-based trial to assess the impact of PATHS 
on children’s academic attainment. It is also one of only a few trials internationally to focus on children 
at the upper end of the primary phase of education—most studies of PATHS have focused on children 
in pre-school and early primary settings (for example the equivalent of the Early Years Foundation 
Stage and Key Stage 1). 


Evaluation objectives 


In late 2011 the evaluators secured funding for a major cluster-randomised controlled trial of PATHS 
from the National Institute for Health Research (NIHR; Project Reference 10/3006/01; ISRCTN number 
85087674). The primary aim of the ‘PATHS to Success’ project was to examine the impact of the 
PATHS curriculum on the social and emotional wellbeing of children aged 7-9 (for example Years 3, 4 
and 5) in English primary schools. A series of secondary outcomes, including health-related outcomes 
and quality of life, were also assessed. The trial included a comprehensive implementation and process 
evaluation (IPE). Forty-five primary schools in the Greater Manchester region were randomly allocated 
to (a) implement PATHS for two years, or (b) continue practice as usual over the same period. 


The main trial protocol can be found at: 


http://www.nets.nihr.ac.uk/__ data/assets/pdf_file/0019/81712/PRO-10-3006-01.pdf 


The EEF agreed to augment the existing trial design with additional outcome assessments focusing on 
children’s academic attainment. This report focuses solely on this aspect of the trial, and there are 
therefore some key divergences between the protocol noted above and the research reported here (for 
example, the primary outcome measures for the main trial are children’s social-emotional skills and their 
mental health, not academic attainment). The findings of the main, NIHR-funded trial will be published 
in due course. 


Our aims were as follows: 


e To assess the impact of PATHS on children’s academic attainment. 

e To determine if PATHS produces ‘differential gains’ in academic attainment for children eligible 
for free school meals (FSM). 

e To assess the extent to which PATHS implementation variability is associated with variability 
in children’s academic outcomes. 
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e To identify process-related issues that might serve to illuminate the mechanisms underpinning 
findings relating to the first three aims. 


Evaluators 


The evaluation team was led by Professor Neil Humphrey, and comprised Dr Alexandra Barlow, Dr 
Michael Wigelsworth, Dr Ann Lendrum, Kirsty Pert, Craig Joyce, Emma Stephens, Lawrence Wo, Dr 
Garry Squires, Professor Kevin Woods, Professor Rachel Calam, Dr Mark Harrison, and Alex Turner. 


Ethical review 


The study was granted ethical approval by the University of Manchester’s University Research Ethics 
Committee (UREC) in 2012 (UREC Ref 11470). 


Eligible schools were recruited using a variety of means, including a one-day conference, the project 
evaluation team’s professional contacts in the region, and letters, telephone calls and follow-up visits 
to individual schools. Headteachers of prospective trial schools were required to sign a Memorandum 
of Agreement (MOA) that outlined key information about the project (Such as the randomisation 
process), what ‘participation’ would entail, and the risks and benefits of taking part. 


Consent to participate was sought prior to randomisation at three levels: school (See above), parent, 
and child. Parental consent was sought using the opt-out method (except in the case of certain aspects 
of the IPE, for example child focus groups, where opt-in consent was utilised). Finally, children were 
required to verbally assent to participation in the research. Sample information and consent sheets can 
be found in Appendix 1. 


Trial registration: NIHR Project Reference 10/3006/01; ISRCTN number 85087674. 
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Methodology 


Design 


The research design was a cluster-randomised controlled trial (Puffer, Torgerson, and Watson, 2005) 
with two arms: intervention (PATHS) and control (usual practice). Schools were the unit of allocation as 
PATHS is a universal intervention—individual or class randomisation was therefore inappropriate (for 
example, class-level randomisation would have introduced considerable risk of contamination). 


The initial project plan was to administer baseline (pre-test) and follow-up (post-test) assessments of 
children’s academic attainment to all children in the target population (Years 3, 4 and 5) in participating 
schools using a standard pre-test—post-test control group design). However, major problems were 
encountered with the administration of the Interactive Computerised Assessment System (InCAS—see 
Outcomes below) that was used as the primary academic outcome measure throughout the trial as well 
as being intended as the pre-test. These problems included insufficient IT resources in schools, 
installation difficulties, assessments taking much longer than the estimated time suggested by the 
developer, and developer server problems. As a result, a post-test only design focusing on children in 
Year 5 (for whom InCAS assessments were administered) and Year 6 (for whom Key Stage 2 academic 
attainment data was available via the National Pupil Database—NPD) was adopted. Gorard (2013) 
notes that a post-test design is ‘generally at least as safe as its alternatives, and is sometimes preferable 
or more feasible than...pre-and-post-test designs’ (p.2). Furthermore, we were fortunate in being able 
to access all participating children’s prior (Key Stage 1) attainment data from the NPD to increase 
statistical power and reduce bias. 


Eligibility 


Mainstream primary schools providing education for children aged 4—11 in the 10 Local Authorities 
(LAs) that make up the Greater Manchester region were eligible to participate. The eligible LAs were 
Bolton, Bury, Manchester, Oldham, Rochdale, Salford, Stockport, Tameside, Trafford, and Wigan. 
Greater Manchester was chosen primarily because of geographical convenience (it is the region where 
the evaluation team is based) but also because of the diversity it provides in terms of urban 
development, socio-economics and other characteristics. Children in Years 3, 4 and 5 on a given 
school’s full-time roll at the start of the 2012-13 academic year were the target population for the study. 
The evaluation team led the recruitment process. 


From an initial pool of 58 schools, 45 met the criteria outlined in the MoA and were randomly allocated 
to either the intervention or control arm of the trial. 


Intervention 


In the interests of clarity, we adapt the Template for Intervention Description and Replication (TIDieR, 
Hoffmann et al., 2014) to describe the PATHS intervention model: 


1. Brief name 
Promoting Alternative Thinking Strategies (PATHS) 
2. Why: Rationale, theory and/or goal of essential elements of the intervention 


PATHS is based on the Affective-Behavioural-Cognitive-Developmental model of development, which 
emphasizes the developmental integration of affect, emotion language, behaviour and cognitive 
understanding to promote social and emotional competence (Greenberg and Kusche, 1993). Core 
programme components are a taught curriculum, generalisation activities and techniques, and parent 
materials. The PATHS programme logic model can be found here: 
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http://www.episcenter.psu.edu/sites/default/files/ebp/Promoting-Alternative-Thinking-Strategies-Logic- 
Model-5-12-2014.pdf 


The model proposes that the implementation of these core components decreases risk factors (Such 
as impulsive behaviour) and increases protective factors (for example school opportunities for pro- 
social involvement), leading to key proximal outcomes of improved knowledge and awareness of 
emotions, self-control, and social problem-solving. Improved academic attainment is viewed as a by- 
product of PATHS implementation. 


3. Who: recipients of the intervention 


All children in a given class: PATHS uses a whole-school ‘universal’ model. The curriculum contains 
lessons for children in Reception through to Year 6. 


4. What: materials 


Curriculum packs were provided for each class containing lessons and send-home activities that 
covered topics such as identifying and labelling feelings, controlling impulses, reducing stress and 
understanding other people's perspectives, in addition to associated physical resources and artefacts 
(such as posters and ‘feelings dictionaries’). Each lesson contains detailed information for the teacher, 
including general and specific objectives, materials required, and procedural notes (including suggested 
wording for teacher input). Sample materials can be viewed at: www.channing-bete.com/prevention- 
programs/paths/preview-intro-paths.html 


Class teachers were also given an implementation guidance manual developed by the evaluation team 
which emphasised the PATHS programme theory and the importance of effective implementation. 
PATHS curriculum packs are distributed in the UK by Barnardo’s. Supplementary materials used in this 
trial (such as the implementation guidance manual) are available on request from the evaluation team. 


5. What: procedures 


PATHS lessons follow a common format that includes an introduction from the teacher (in which the 
lesson topic and objectives are introduced), a main activity (often built around a group activity or story), 
and a brief plenary/closure (in which learning is reviewed). Frequent prompts to elicit pupil responses 
and clarify learning are included throughout. 


The programme utilises a ‘spiral’ curriculum model whereby (i) topics and concepts are revisited; (ii) 
units and lessons are developmentally sequenced; (iii) new learning is linked to previous learning; and 
(iv) the competence of learners increases with each successive visit to a topic or concept. 


6. Who: implementers of the intervention 


PATHS was implemented by class teachers in Years 3, 4 and 5 (Years 4, 5 and 6 in the second year 
of the trial). All were qualified teachers and had an average of eight years teaching experience; 81% 
were female. 


7. How: mode of delivery 


PATHS lessons were delivered as part of the normal class timetable, typically in the slot(s) allocated 
for Personal, Social and Health Education. This meant that PATHS did not displace any core curriculum 
subject time such as English, Maths, or Science. Generalization activities and strategies were to be 
implemented routinely throughout the school day. 


8. When and how much: dosage 
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Lessons lasted approximately 30-40 minutes and were to be delivered twice-weekly throughout the 
school year. Curriculum packs contained an average of 40 lessons. The recommended dosage for 
PATHS would be the delivery of most or all of these lessons in a year. 


9. Tailoring: adaptation of the intervention 


PATHS is a ‘manualised’ intervention (prescriptively defined in a manual), and optimal implementation 
fidelity is emphasized by the programme developers. Nonetheless, implementers are encouraged to 
make surface adaptations (for example changes of names in stories) in order to increase a sense of 
ownership and fit to local context. 


10. Modifications 


As noted above, the evaluation team developed an implementation guidance manual to supplement the 
materials provided by Barnardo’s. 


11. How well: planned 


Strategies to maximise implementation effectiveness included one full day of initial training for teachers 
with a half-day follow-up four months later. Training was led by staff from the Evidence-based 
Prevention and Intervention Support Centre at Pennsylvania State University (PSU). This was 
supplemented by technical support and assistance (for example lesson modelling, observation and 
feedback) from three members of the evaluation team, who were themselves trained by PSU staff and 
received ongoing supervision throughout the trial. 


12. How well: actual 


A range of aspects of implementation (Such as fidelity/adherence, dosage, quality, and participant 
responsiveness) were assessed formally through structured observations of lessons conducted 
throughout the trial by members of the evaluation team as part of our IPE. Mean scores—on a scale of 
1 to 10—for fidelity (8.20), quality (8.48), participant responsiveness (7.34) and reach (9.08) were all 
high. However, mean dosage scores indicated that classes were an average of 20 lessons behind 
schedule (the equivalent of 10 weeks at 2 lessons per week) at the point of observation. Thus, PATHS 
lessons were generally implemented well, but not at the frequency recommended by the programme 
developers. 


‘Usual practice’ was assessed in al! schools (whether intervention or control) as part of the IPE, both 
prior to randomisation and at 12-month follow-up, via school-level surveys. Usual practice comprised a 
range of named initiatives including (but not limited to): the whole-school component of the social and 
emotional aspects of learning (SEAL) programme; Circle Time; the National Healthy Schools 
programme at the universal level; the Targeted Mental Health in Schools programme; nurture groups; 
and targeted components of the SEAL programme at the indicated/targeted level. Although it is a non- 
statutory subject, participating schools also taught children personal, social and health education. 


Outcomes 


Year 5 


For children in Year 5 at the point of post-test, INCAS (developed by the Centre for Evaluation and 
Monitoring—CEM) was used to generate outcome data relating to attainment in reading and maths. 
InCAS was used as it was recommended by the EEF as a suitable independent measure of academic 
attainment, and was designed for use with children aged 6—11. Further information about INCAS can 
be found at the developer’s website: http://www.cem.org/primary 
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InCAS assessments were administered by members of the evaluation team. Scoring was undertaken 
by CEM, blind to the allocation status of individual schools and children. 


Year 6 


For children in Year 6 at the point of post-test, Key Stage 2 national curriculum assessments were used 
to generate outcome data relating to their attainment in English and Mathematics. These assessments 
were used as they offer optimal external validity (providing the metric by which the academic progress 
of children is judged as they reach the end of primary education in England) and no additional data 
burden for participating schools (since the tests are statutory). Testing was undertaken by school staff 
following the government's recommended protocol (Standards and Testing Agency, 2014). Scoring was 
undertaken by the Standards and Testing Agency, blind to the allocation status of individual schools 
and children. This data was subsequently extracted from the NPD. 


Sample size 


Power and sample size (PASS) calculations for the main NIHR-funded study were based on the primary 
outcome of children’s social and emotional competence and are detailed in the trial protocol (see link 
above). In view of the issues noted earlier and the ensuing changes to the study design, PASS 
calculations provided here are necessarily post-hoc. In the case of both cohorts, calculations utilise a 
prior attainment and demographic covariates model (Hedges and Hedberg, 2007). Prior academic 
attainment data from end of Key Stage 1 national curriculum assessments were available from the NPD. 
Demographic covariates were sex and FSM eligibility. School-level minimisation variables (for example 
FSM and English as an Additional Language—EAL) were also included in the calculation of the intra- 
cluster correlation coefficients (ICCs) noted below. 


Year 5 


There were 1,705 Year 5 children in the 45 participating schools. After accounting for the covariates 
specified above, the sample ICC was 0.08. With an average of 38 children per cluster, an average 
correlation of 0.71 between the prior attainment and post-test data, and Power and Alpha set to 0.8 and 
0.05 respectively, the minimum detectable effect size (MDES) was determined to be 0.19. 


Year 6 


There were 1,631 Year 6 children (PATHS n = 847; usual practice n = 784) in the 45 participating 
schools. After accounting for the covariates specified above, the sample ICC was 0.04. With an average 
of 36 children per cluster, a correlation of 0.73 between the prior attainment and post-test data, and 
Power and Alpha set to 0.8 and 0.05 respectively, the MDES was determined to be 0.17. 


Randomisation 


Schools were randomly allocated to the intervention and control arms of the trial independently of the 
evaluation team by the Manchester Academic Health Science Centre Trials Co-ordination Unit 
(MAHSC-CTU). The randomisation procedure incorporated a minimisation algorithm to ensure balance 
across the arms of the trial in terms of the proportions of children eligible for FSM and speaking EAL in 
participating schools. 


Analysis 


Outcome data was analysed using hierarchical linear modelling (also known as ‘multi-level modelling’) 
in MLWin Version 2.32 in view of the clustered and hierarchical nature of the datasets. Three-level 
(school, child, time) models were constructed, with allocation status (PATHS vs. usual practice) and 
minimisation variables entered at the school level, sex and FSM eligibility entered at the child level, and 
academic outcome data (prior attainment and post-test) entered at the time level. We utilised both 
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intention to treat (Gupta, 2011) and sub-group (Petticrew et al., 2012) analyses, the former providing 
an estimate of the overall impact of PATHS on children’s academic attainment and the latter allowing 
us to identify any differential gains experienced by children eligible for FSM. All continuous data was 
standardised prior to analysis to facilitate comparison of ES (Cohen's d) within and across models. All 
ES were then converted to Hedge’s g as per EEF specifications. 


MLWin by default incorporates all available data (for example it includes data for participants with and 
without complete datasets) in its model fitting using full information maximum likelihood (FIML) 
estimation procedures under a random dropout assumption (where ‘missingness’ is assumed to depend 
on observed measurements). FIML has been shown to be an effective method in handling missing data 
(Twisk, 2006; Twisk and de Vente, 2002) and is considered by some to be superior to multiple 
imputation techniques as it correctly estimates standard errors (Larsen, 2011). 


Implementation and process evaluation (IPE) methodology 


A process evaluation was not commissioned as part of this EEF study. However, as noted earlier, the 
main NIHR-funded trial included a comprehensive IPE, from which some relevant data is reported here. 


Briefly, the IPE comprised: 


I. Surveys of usual practice in all participating schools at baseline and 12-month follow-up. These 
surveys assessed each school’s involvement and implementation status for a wide range of 
both universal and targeted/indicated SEL interventions, and were completed by each school’s 
Key Stage 2 co-ordinator. 

ll. Structured observations of PATHS lessons in intervention schools to assess fidelity/adherence 
(for example, to what extent were the prescribed activities in the lesson replicated?), dosage 
(for example, what lesson was being delivered relative to the planned delivery schedule in the 
implementation manual?), quality (for example, how well prepared, enthusiastic, engaging and 
participatory is the teacher’s delivery of the lesson?), participant responsiveness (for example, 
to what extent do children in the class engage with the lesson?), reach (for example, what 
proportion of the class are present in the lesson?) and adaptations (for example, what is the 
nature and extent of changes made to the lesson by the teacher?). The observation schedule 
was adapted from existing rubrics in other studies. It was piloted and refined using video 
footage of lessons being implemented in schools in the aforementioned Birmingham trial (Little 
et al., 2012). The observations were conducted by three members of the evaluation team (each 
observing in schools where they were not providing technical support and assistance in order 
to minimise bias), and 10% were moderated by our IPE lead. In view of the use of multiple 
observers, the aforementioned video footage was also used to help ensure consistency. 

ll. Teacher self-report implementation surveys in intervention schools to assess 
fidelity/adherence, dosage, quality, participant responsiveness, and reach. These surveys were 
developed to mirror the observation schedule described above. 

IV. Teacher self-report surveys to assess a range of factors affecting implementation, including 
preplanning and foundations, implementation environment, implementation support system, 
implementer characteristics, and intervention characteristics. These surveys were all adapted 
from existing measures. 

V. Semi-structured interviews with teachers and other members of school staff (Such as 
headteachers) in intervention schools to explore implementation and process issues (for 
example the acceptability and feasibility of PATHS). 

Vi. Focus groups with children in intervention schools to explore their experiences of PATHS. 

Vil. Interviews with parents to assess the extent of their awareness of, involvement in, and attitudes 
towards PATHS. 
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Of the above, we utilise (I), (II) and (V) in this report as these were felt to provide the most useful 
explanatory data (and, of course, we experienced natural restrictions on what could be included 
because of reporting deadlines). 


Costs methods 


We report the ‘real world’ costs of PATHS—those that would be experienced by a typical primary school 
outside of the context of a major randomised trial. These costs were provided by the PATHS team at 
Birmingham City Council as it provides both materials and implementation support for PATHS as part 
of a traded service with English schools. The basic unit cost per pupil in the first year of implementation 
was calculated by dividing the overall cost by an estimated 189 pupils (that is, seven classes of 27 
children from Reception to Year 6). 
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Impact evaluation 


Timeline 


School recruitment took place from January to April 2012. Baseline assessments for the main trial took 
place in May, June, and July 2012. Following the completion of the baseline, schools were randomised 
by MAHSC-CTU. Teachers in intervention schools received their initial PATHS training in early 
September 2012 (with the half-day follow-up in January 2013). Implementation began immediately after 
the initial training. In the second year of the trial, training was provided for any new teachers in the target 
year groups and those responsible for Year 6. Post-test assessments took place in May, June, and July 
2014. 


Participants 


All mainstream primary schools in the 10 LAs in Greater Manchester were invited to participate in the 
project in January 2012. Initially schools were sent email and postal invitations to a recruitment event 
in March 2012: 35 schools attended, of whom 19 signed up for the trial. In parallel, schools were sought 
via the team’s professional contacts in the region (Such as LA staff), and letters, telephone calls, and 
follow-up visits were made to individual schools. These methods secured the remainder of the initial 
recruitment sample. 


Schools were randomly allocated to the intervention and control arms of the trial, independently of the 
evaluation team, by the Manchester Academic Health Science Centre Trials Co-ordination Unit 
(MAHSC-CTU). The randomisation procedure incorporated a minimisation algorithm to ensure balance 
across the arms of the trial in terms of the proportions of children eligible for FSM and speaking English 
as an additional language. 


A diagram illustrating the flow of participating schools and children through the main study is provided 
below (Figure 1). In the interests of clarity we provide separate flow information for the two cohorts 
(Year 5 and Year 6) because of the different attrition rates at school and pupil levels (due to, for 
example, school drop-out or pupil absence). For the Year 6 cohort (N = 1,631), there was zero attrition 
at the school level and 3% (N = 49) at the pupil level. For the Year 5 cohort (N = 1,705) there was 16% 
(N = 268) attrition at the school level and 19% (N = 320) at the pupil level. 
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Figure 1: Flow of schools and children through the main trial. 


Recruitment 
(Jan-Apr 2012) 


bod Tole) (Mi aclergtlic-te| 


SYoi stele) cfe—mots: 


Baseline and randomisation 
(May-July 2012) Tod stole) M1 Ife] 1s) (-Miels 
re-Taleleyealtcy-lale) a] 


13 schools failed to complete the baseline 
measures for the main NIHR trial, because of: Schools = 45 
e lack of time (n=7) Year 5 N = 1705 
* lack of IT facilities (n=2) Year 6 N = 1631 
* change in staff (n=2) 
¢ other priorities (e.g. OFSTED) (n=1) 
¢ lack of response (n=1) 
PATHS Usual practice 


Year 5 Year 5 
Schools = 23 Schools = 22 
IN Ea stolo) N = 850 
Year 6 Year 6 
Schools = 23 Schools = 22 
N = 847 N = 784 


Post-test 
(May-July 2014) PATHS UFSITE Lo) g-To ter -) 
Year 5 cohort Year 5 Year 5 
Schools = 22 Schools = 15 
8 schools (268 pupils) lost to N = 623 N = 494 
follow-up because of: Year 6 Year 6 
* change in staff (n=2) To! slele) | ieaw20) To} plete) ia 4 
lack of response (n=2) N = 823 N= 759 


¢ other priorities (n=1) 
* non-compliance with InCAS (n=3) 


320 pupils lost to follow-up because of absence, changing schools, not sitting test etc 
Year 6 cohort 
0 schools lost to follow-up 


49 pupils lost to follow-up because of absence, changing schools, not sitting test etc 


Table 1 summarises the characteristics of the schools in each arm of the trial, with national averages 
provided as a reference point against which to assess the representativeness of the sample. The school 
sample characteristics mirror those of primary schools in England well, with the exception of being 
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somewhat larger and having higher proportions of pupils eligible for FSM and speaking EAL. This 
probably reflects the fact that project was more likely to appeal to large, ethnically diverse urban schools 
in disadvantaged areas, where the perceived need for SEL is typically greater. In terms of balance, the 
difference between intervention and control schools was negligible across all characteristics with the 
exception of attainment and attendance (both small ES using Cohen's (1992) criteria, although 
attendance was on the cusp of a medium ES). 


Table 1: School sample characteristics and national averages 


Yod aColo) Med at-Ter-(eaclaloiile National PATHS Usual Balance at 
average sample practice re-Varocodsalecy-lacey a} 
ri=x-1 0M (12) rut=x- 10 (=) 2) ee (=) 


Size'—number of full-time 233.4 313.26 287.36 
equivalent pupils on roll (111.15) (96.47) 


Attendance*—overall absence (% 5.2 5.06 5.60 
half-days missed) (0.94) (1.24) 


FSM‘*—proportion of pupils : 30.13 30.86 
eligible for free school meals (20.12) (19.29) 


EAL'—proportion of pupils 17.5 20.63 23.55 
speaking English as an additional (24.65) (24.51) 
language 


SEND*—proportion of pupils with 19.8 16.50 17.28 
special educational needs and (5.99) (5.96) 
disabilities 

Attainment*—proportion of pupils 75 80.73 75.38 
achieving level 4 (or above) in (12.09) (11.88) 
English and Maths at end of KS2 


1 Department for Education (2010), ? Department for Education (2014), * Department for Education (2012), 4 Department for 
Education (2013). 


Pupil characteristics 


Table 2 summarises the characteristics of pupils in each arm of the trial, with national averages provided 
as a reference point against which to assess the representativeness of the sample. Our pupil sample 
characteristics mirror those of primary schools in England closely, albeit with a similar pattern of 
deviation in terms of FSM and EAL to that noted above. In terms of balance on key observables, the 
difference between intervention and control pupils in relation to both eligibility for FSM (ES = 0.08) and 
prior attainment (ES = 0.03) was negligible. All other observables were considered to be well balanced 
at randomisation. 


Table 2: Pupil sample characteristics and national averages 


Pupil characteristic National PATHS Usual Balance at 
average sample practice re-Valokeynalicy-laceya} 
mean (SD) mean (SD) (=) 


Sex—proportion of male students 50 49.9 53 0.08 


FSM'—proportion of pupils eligible 28 31.6 28.6 0.08 
for free school meals 


EAL'—proportion of pupils 
speaking English as an additional 
language 
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Mental health2—mean SDQ Total 6.7 7.72 7.43 
Difficulties (teacher-rated) subscale (2.46) (2.42) 
score 


Prior attainment—KS1 average 15.24 15.12 
points score for English/Maths (3.78) (3.85) 


+ Department For Education (2010), ? www.sdginfo.org 
Outcomes and analysis 


Table 3 summarises the main findings of our analyses. Full details of the models can be found in 
Appendix 2. 


Table 3: The impact of PATHS on pupils’ academic attainment—key findings. 


Post-test means Effect size 
PATHS Usual practice 
Outcome | Mean N Mean Hedge’s Months Statistical 
(missing) (95% Cl) (missing) (95% Cl) g (95% progress significance 
re) 
Year 5 623 (232) 97.83  494(356) 98.35 0.026 1 x 
Maths (96.31 to (96.65 to —(-0.08 to 
(InCAS) 99.36) 100.04) 0.13) 
Year 5 642 (213) 100.53 492 100.98 -0.029 -1 x 
Reading (99.49 to (358) (99.87 to (-0.13 to 
(InCAS) 101.57) 102.10) 0.07) 
Year 5 166 (88) 90.19 115(113) 86.61 -0.036 -1 x 
Maths © (87.67 to (83.67 to (-0.24 to 
(INCAS) if 92.70) 89.55) 0.17) 
FSM 
eligible 
Year 5 167 (87) 95.17 114(114) 93.50 0.017 0 x 
Reading (92.97 to (91.01 to (-0.18 to 
(INCAS) if 97.37) 95.99) 0.22) 
FSM 
eligible 
Year 6 823 (24) 29.11 759 (25) 28.77 -0.025 -1 x 
Maths (28.75 to (28.39 to (-0.012 to 
(KS2) 29.48) 29.14) 0.07) 
Year 6 823 (24) 28.73 759 (25) 28.52 -0.106 -2 J 
English (28.45 to (28.21to (-0.19 to 
(KS2) 29.01) 28.83) 0.02) 
Year 6 261 (5) 27.46 202 (2) 26.55 0.016 0 x 
Maths (26.80 to (25.86 to (-0.015 to 
(KS2) if 28.12) 27.25) 0.19) 
FSM 
eligible 
Year 6 261 (5) 27.28 202 (2) 26.70 0.120 +2 x* 
English (26.73 to (26.08 to (-0.03 to 
(KS2) if 27.83) 27.32) 0.27) 
FSM 
eligible 


*Marginal, non-significant trend (for example p<0.10) 


As can be seen from the above table, the overall trend is that PATHS failed to impact upon pupils’ 
attainment. This is generally consistent across year groups (Year 5 and Year 6), outcome measures 


Education Endowment Foundation 17 


Promoting Alternative Thinking Strategies 


(INCAS and KS2 assessment), subject areas (maths and English/reading), and analytical frames 
(intention to treat and sub-group). The only exceptions are an effect on KS2 English scores at intention 
to treat favouring the control group and a marginal, non-significant effect (p = 0.06) on KS2 English 
scores for FSM eligible pupils favouring the PATHS group. However, in both cases the ES are 
somewhat modest, equating to only 2 months progress, and so are unlikely to be ‘practically significant’ 
(Hill, Bloom, Black, and Lipsey, 2008). Thus, in relation to our first study aim outlined at the beginning 
of this report, PATHS did not appear to have any impact on children’s academic attainment. With regard 
to our second study aim, PATHS did not appear to produce ‘differential gains’ in academic attainment 
for children eligible for FSM. 


Cost 


The cost of PATHS in the context of this trial differs significantly from the ‘real world’ costs of the 
programme because (a) intervention costs were funded by the EEF rather than schools, (b) a ‘bulk 
discount’ was applied to the intervention materials because a large number of curriculum packs were 
ordered, and (c) training was carried out en masse, with coaching support being provided free of charge. 
Thus, below we provide the typical costs for a school under normal circumstances. The figures 
presented were provided by the PATHS team at Birmingham City Council as it provides both materials 
and implementation support for PATHS as part of a traded service with English schools. 


For a single form entry school, the initial cost of PATHS is £6,532. This ‘bundle’ figure includes the 
essentials for PATHS implementation: a curriculum pack for each year group from Reception to Year 
6; additional/supplementary materials (for example books, CDs, puppets, ‘feeling faces’ and posters); 
initial training for seven teaching staff; headteacher and/or PATHS co-ordinator training; and two days 
of coaching time. Each of the different components included in this bundle are also available separately 
(for example, the Reception—Year 6 curriculum pack is available for £2,100, with individual year group 
packs costing between £295 and £515). Additional coaching is charged at £250/£500 per half-day/day. 
Finally, supplementary and/or advanced courses are available (for example a PATHS awareness 
course for parents costs £2,200). 


Thus, for a single form entry school with seven classes with an average of 27 pupils (189 pupils on roll), 
the cost per pupil in the first year of implementation—for schools adopting the bundle cost above—is 
£6,532/189 = £34.56. Obviously, this figure reduces in each subsequent year as most of the above 
items are one-off, non-recurrent costs. Thus, in the eighth year of implementation—at which point a 
total of 378 pupils (the original cohort of 189 plus a new intake of 27 per year) would have participated— 
the cost per pupil is £6,532/378 = £17.28. However, there are recurrent costs associated with additional 
coaching and/or training days should schools wish to make use of the external support model 
recommended by the PATHS developers. For schools seeking only the PATHS curriculum packs 
themselves, the initial cost per pupil is £2,100/189 = £11.11. By the eighth year of implementation, this 
drops to £2,100/378 = £5.56. 


In terms of teacher time, basic training is 1.5 days (1 initial day, 0.5 day follow-up), or approximately 
10.5 hours. Each lesson is assumed to incur a relatively small amount of planning and preparation time 
given the prescriptive manual-driven nature of the intervention—approximately 20 minutes to allow 
familiarisation and preparation of materials. Each lesson takes on average 30 minutes to deliver. Thus, 
a single PATHS lesson uses around 50 minutes of teacher time. Taking Year 5 as an example, the 56 
lessons (44 plus 12 ‘Jump Start’ lessons recommended for use with classes that have not previously 
taken part in PATHS) would incur around 57 hours of teacher time in the first year of implementation if 
training was attended and all lessons were delivered. Obviously, this figure is approximate and does 
not include activities which are much more variable (for example time spent engaging with coaches, 
and generalisation activities). 
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Implementation and process evaluation 


As noted earlier, a process evaluation was not commissioned as part of this EEF study, however we 
are able to draw upon certain elements of the comprehensive IPE conducted as part of the main trial. 


Implementation variability and its association with pupil outcomes 


In the first section of this chapter we present quantitative analyses in which we examine the associations 
between different aspects of PATHS implementation and children’s academic outcomes. Here we use 
data drawn from our independent, structured observations as this is generally considered a more 
rigorous and valid means through which to assess an implementation rather than teacher self-report 
(which can be prone to social desirability and/or impression management effects—see Domitrovich et 
al., 2008; Domitrovich, Gest, Jones, Gill, and DeRousie, 2010). However, the use of such observational 
data comes with its own limitations, chief amongst which is that it necessarily captures only a ‘snapshot’ 
of the implementation process. 


Given that there are no agreed thresholds for implementation ratings (for example, it is not possible to 
definitively say what counts as an ‘acceptable’ level of fidelity/adherence), we used the observational 
data to grade each class/teacher as either ‘low’, ‘moderate’ or ‘high’ for each aspect of implementation 
according to where their raw scores sat in the overall distribution—low, < -1 SD; moderate, -1 to +1 SD; 
and high, > +1 SD. The only exception to this was reach: this was coded as high (100% of pupils 
present), moderate (90-99%), or low (< 90%) according to the proportion of pupils present during the 
PATHS lesson being observed. For all aspects of implementation, ‘low’ was set as the reference group 
in the models noted below. 


As already noted, mean scores (minima 1; maxima 10) for fidelity (8.20), quality (8.48), participant 
responsiveness (7.34) and reach (9.08) were all high. However, mean dosage scores indicated that 
classes were an average of 20 lessons behind schedule at the point of observation (the equivalent of 
10 weeks at 2 lessons per week). Thus, PATHS lessons were generally implemented well, but not at 
the frequency recommended by the programme developers. 


As in the assessment of outcomes reported in the previous chapter, data was analysed using 
hierarchical linear modelling (also known as ‘multi-level modelling’) in view of the clustered and 
hierarchical nature of the datasets. Four-level (school, class/teacher, child, and time) models were 
constructed: minimisation (for example FSM) and usual practice (for example use of universal and 
targeted SEL approaches) variables were entered at the school level; implementation variables (for 
example fidelity/adherence, quality, dosage) were entered at the class/teacher level; sex and FSM 
eligibility were entered at the child level; and academic outcome data (prior attainment and post-test) 
was entered at the time level. As with the assessment of outcomes, continuous data was standardised 
to facilitate comparison of ES within and across models. 
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Table 4 summarises the main findings of these analyses. Full details of the models can be found in 
Appendix 3. 


Table 4: PATHS implementation variability and its association with academic outcomes 
Y5 English Y5 Maths Y6 English Y6 Maths 


ES 


Schoo Usual practice x x x* 
Ilevel (universal) 


Usual practice 
(targeted) 


Fidelity 0.032 (if x 0.021 (if x -0.177 (if -0.104 (if 
(compared to moderate moderate moderate moderate 


low) ) ) ) ) 


0.316 (if -0.140 (if 0.031 (if -0.036 (if 
high) high) high) high) 


Dosage -0.032 (if -0.050 (if -0.001 (if -0.069 (if 
(compared to moderate moderate moderate moderate 


low) ) ) ) ) 


0.000 (if 0.000 (if 0.000 (if 0.000 (if 
high) high) high) high) 


Quality 0.109 (if 0.065 (if 0.691 (if 0.682 (if 
(compared to moderate moderate moderate moderate 


low) ) ) ) ) 


0.315 (if 0.066 (if 0.622 (if 0.695 (if 
high) high) high) high) 


Reach 0.323 (if 0.341 (if 0.458 (if 0.298 (if 
(compared to moderate moderate moderate moderate 


low) ) ) ) ) 


-0.178 (if -0.212 (if 0.151 (if 0.190 (if 
high) high) high) high) 


Participant 0.180 (if 0.301 (if -0.345 (if -0.422 (if 
responsivenes moderate moderate moderate moderate 
s (compared to) ) ) ) 
low) 
0.000 (if 0.000 (if -0.123 (if -0.261 (if 
high) high) high) high) 


*Marginal, non-significant trend (for example p < 0.10) 


There are several themes worthy of note here. First, variability in usual SEL practice at the targeted 
level, implementation quality, reach and participant responsiveness were each associated with outcome 
variability in one or more of our analyses. This speaks to the need for a comprehensive approach to the 
assessment of implementation (Lendrum, Humphrey and Greenberg, 2016). Second, in most cases, it 
is moderate rather than high (compared to low) levels of implementation that appear to make a 
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substantive difference. This is consistent with some previous research in the field of implementation 
science (for example Durlak and DuPre, 2008). Third, where findings are statistically significant, the ES 
associated with implementation variability are of a magnitude that is likely to be practically significant, 
for example a 0.691 SD increase in Year 6 English scores is associated with moderate (compared to 
low) quality implementation. 


Implementation fidelity and dosage of PATHS appear to be unrelated to academic outcomes. Given the 
extensive literature base supporting these two aspects of implementation as moderators of intervention 
outcomes, this was somewhat surprising. However, many previous studies have examined only fidelity 
and dosage, and so it may be that the more comprehensive approach taken here has revealed a more 
complete picture (for example, the influence of fidelity may be greatly lessened once we have taken 
into account the actual quality of delivery). Also, much less is known about how implementation 
variability influences distal, as opposed to proximal, intervention outcomes (thus, fidelity and dosage 
variability may be more closely associated to outcomes such as children’s social and emotional skills). 
Finally, in relation to dosage, it is important to note that classes/teachers were, on average, 20 lessons 
behind schedule. The nature of the exposure—response relationship may be such that none of the 
classes had received the sufficient number of lessons for us to be able to detect a dosage effect (a point 
which we return to later). 


Quality also appears to be a key moderator of student outcomes, producing the largest ES across the 
four implementation analyses for Year 6 English and Maths respectively (See Figure 2). Taken in 
tandem with the (null) fidelity findings reported above, it would appear that teacher preparedness, 
enthusiasm, and the manner in which they engage children in PATHS lessons are ultimately more 
strongly associated with children’s attainment than the extent to which they adhered to the lesson 
scripts. This is an important message given the dominant discourse of fidelity, particularly in 
‘manualised’ (prescriptively documented) interventions such as PATHS. 


Of all the aspects of implementation assessed in our observations, reach was the only one associated 
with outcome variability across all four analyses. To a certain extent this is self-explanatory—an 
intervention can only trigger change if the intended recipients are present when it is being delivered 
(particularly given the relatively meagre dosage levels noted above). Finally, participant responsiveness 
produced an unexpected finding, with an inverse association with Year 6 Maths scores being found 
(indicating that where children were rated as moderately responsive in PATHS lessons (compared to 
the low group), this was associated with a 0.422 standard deviation decrease in their Maths scores). 


Figure 2: Year 6 English SAT scores and PATHS implementation quality 
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Control group activity 


As noted earlier in this report, our assessment of usual practice extended to both arms of the trial and 
data was available via school-level surveys prior to randomisation and at one-year follow-up. ‘Usual 
practice’ in the control group comprised a range of named initiatives including (but not limited to): the 
whole-school component of the social and emotional aspects of learning (SEAL) programme; Circle 
Time; and the National Healthy Schools programme at the universal level; and the Targeted Mental 
Health in Schools programme; nurture groups; and the targeted component of the SEAL programme at 
the indicated/targeted level. Our analyses indicated that compared to those implementing PATHS, 
schools in the control arm had significantly increased their SEL-related activities at both the universal 
(ES = 0.42, p < 0.05) and targeted (ES = 0.52, p < 0.05) levels at the one-year follow-up stage. This so- 
called ‘John Henry effect’ (also known as compensation rivalry) is seen as a direct response to being 
randomly allocated to the usual practice arm and poses a potential threat to validity. However, re- 
analysis of the outcome data, presented in Table 3, that incorporated this data in the modelling 
procedures revealed no substantive differences in outcomes. That is, the changes in control group 
behaviour observed had no bearing on the failure of PATHS to improve children’s academic outcomes 
relative to the control group. 


Selected process evaluation findings 


Here we provide a brief summary of selected findings from our qualitative exploration of the 
implementation and process-related issues that were evident in PATHS schools. This section is 
deliberately highly selective and purposive in nature. We present data that can help to illuminate and 
explain the findings reported earlier in this report. Data excerpts in the following text are from teacher 
interviews. 


Philosophical fit 
Staff interview data yielded useful insights regarding the cultural transferability of PATHS. This included 


perceptions of the programme’s philosophical fit (for example, congruence with community, school, and 
teacher values/norms), social validity (for example, the extent to which it met a perceived need within 
the school), practical fit (for example, organisational and classroom requirements, pedagogical style, 
appropriate resources) and issues relating to implementer understanding and skills (for example, 
teachers’ understanding of the principles of SEL). In relation to philosophical fit, the approach 
underpinning PATHS appeared to be compatible with the ethos of most of the intervention schools: ‘We 
believe in delivering a whole curriculum, not just the academic subjects, to our children. We want them 
to be good social individuals when they leave our school’. Certain aspects of the programme were 
particularly well received, including the ‘Golden Rule’, in which pupils are taught about the maxim, ‘treat 
others as you would expect to be treated yourself’: ‘That’s like my philosophy on life’. 


Meeting perceived needs 
For many, PATHS was seen as meeting a clear need within school: ‘I think because of contextual 


background there are a lot of children with a lot of varying social and emotional needs, and issues with 
understanding how to deal with anger or frustration. They may not have the verbal skills, they may not 
have the role models that some people have’. In particular, several teachers noted the utility of PATHS 
for children with difficult or chaotic home lives: ‘There are a lot of children here with problems outside 
of school and they will have had issues... it’s been very good in that way’. However, this view was by 
no means universal, and some staff questioned the extent to which a whole school approach—that they 
saw as benefiting only a few pupils—was justified (‘It’s obviously more useful for children who need it 
more... but when you don’t have that many issues with them anyway it’s harder to keep bringing it up 
because you're not needing it’), or indeed whether it was actually an effective model for vulnerable 
children (‘It doesn't work for the kids who need it just because they're too much into a raw emotion, 
sometimes they can’t even explain what the issue is’). 


Practical fit—curriculum and implementation issues 
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In terms of practical fit, the approach schools and teachers took to integrating PATHS into the timetable 
had a clear bearing on their attitudes and approach to implementation: 


‘It’s replaced our PSHE, so it means we don’t discuss the wider PSHE issues anymore like global 
warming... that kind of stuff is basically just gone... | would like a bit more balance so maybe if it could 
be like that [PATHS] once a week and then PSHE once a week just to give that balance again’. 


The PATHS materials themselves produced mixed reactions from teachers, with some praising its 
developmental appropriateness (‘I think they have been very appropriate... it seems to be about the 
right pitch for their age’) while others felt that it was insufficiently anglicized (‘It’s too Americanisea’, 
‘Some of the things in the booklet are very American’). As a result, adaptations were common, but were 
primarily ‘surface’ changes to improve fit with pupils’ needs, context, and/or individual teaching styles: 
‘| will change it into my own words that | know would fit more with my children... but still making sure 
the elements are the same’. Although less frequent, some teachers reported making deeper changes, 
such as collapsing two lessons into one, or skipping lessons or concepts that they felt were 
inappropriate: 


‘Like the Islamic population of the school really. Not that it’s a taboo thing, it’s just... we just decided not 
to do that one. We just skipped that one. We thought maybe we needed to give it a bit of time and come 
up with another problem that wasn’t that. And I think... it could have got misconstrued if children had 
gone back saying we did a story about children drinking alcohol. It isn’t that but parents wouldn't 
understand. ’ 


Pedagogical fit 
Teacher responses regarding the pedagogical fit of PATHS centred primarily on the fact that it is a 
‘manualised’ intervention. For some, this high level of prescription was comforting: 


‘It’s very simple to follow... and you don't really have to prepare anything.’ ‘It’s straightforward, it’s very 
basic common sense really... the objectives are very clear and... you don't have to worry about making 
resources.’ 


But for others it was constraining and not conducive to pupils’ engagement: 


‘The plans have far too much detail in and there’s a lot of reading to get to the bare bones of what the 
lesson’s doing... quite often they [the pupils] kind of sit there and however many strategies you use 
they go “Oh its PATHS, we’re going to get a story, we’re going to talk about it, we’re bored of it”.’ 
There were concerns about the length of some lessons, and many teachers reported that they would 
prefer a more interactive approach (‘/ think sometimes the lessons are really, really long... | can’t teach 
too much from a script’). Despite these concerns, there was a clear sense that PATHS lesson objectives 
were directly relevant to key aspects of the teacher’s role, such as classroom management, or were 
useful as a tool for responding to individual incidents: ‘I like doing it with them because a /ot of it is really 
relevant, the lessons that will come up, | will have had an incident happen linking to that... so | can see 
the value of it’. However, some teachers struggled to fully embed PATHS into their existing behaviour 
management strategies: 


‘You have to embed it throughout for it to be truly effective. | think what we're lacking is implementing it 
all the time. So referring to it daily within your behaviour rules... I’m just talking from my class really 
because | have my own behaviour strategies that | often forget to relate to the PATHS, to embed that 
in. So I'd say they, to really adopt it fully. Adopt it fully, take it on and make sure they're using it all the 
time.’ 


Most teachers were highly familiar with the underlying principles and concepts of PATHS (‘It’s common 
sense really’). The trial followed on from a period of approximately six years in which the SEAL 
programme had been established in most primary schools across England. As a result, there was a 
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strong foundation in place already: ‘I think because I’ve done SEAL... it’s kind of familiar because I’ve 
been doing it for a long time’. However, for some, the impression was that PATHS was perhaps 
insufficiently distinctive: ‘It seems to cover a lot of the SEAL aspects that we were doing before’. 


Barriers and facilitators to effective implementation 

Our process evaluation revealed explicit barriers to, and facilitators of, effective implementation that 
may help to account for the findings. At the macro level, an increasing shift toward a ‘rationalist’ model 
of education under the current Coalition government meant that many struggled to prioritise PATHS: 


‘With all the pressure that we’re under to have children achieve in English, Maths and Science... I’d 
much rather somebody who was a whizz at Maths could come in and say: “that's not working, let’s try 
this”. It’s obviously not as important as literacy, numeracy, and things like that—and we’ve got a huge 
focus on reading this year. It’s there, but | don’t think it’s a huge focus.’ 


Even in cases where PATHS was viewed as critical, instruction in core academic curriculum areas 
always won out (‘Well obviously it’s massively important, but in terms of literacy and numeracy... it is 
below that’). As one teacher noted: ‘something has to give’. 


The issue of prioritisation often resulted in a perceived lack of time in the timetable to teach PATHS 
lessons at the recommended frequency (‘We have introduced a new literacy curriculum which takes up 
[lots of time] and it’s very, very inflexible’). Other initiatives (for example ‘We are going for the Rights 
Respecting Award... so there hasn't really been a focus on PATHS as a whole school at the moment’) 
or local, contextual issues (‘We’re a church school so we have to do two and a half hours of RE’) 
regularly took precedence over PATHS. Indeed, even routine school events and processes were 
perceived to affect the consistency of PATHS delivery: ‘You’ve got the performance things for Easter, 
and the summer term one, Christmas, and harvest, and when you get rid of all these blocks you're 
trying to fit it in your timetable’. These practical issues meant that, despite their best intentions, most 
teachers were unable to deliver PATHS on a twice-weekly basis: 


‘The intention was twice a week, but the reality of it was very different. Twice a week is nigh on 
impossible with everything else. I’m sure you've heard that from everybody. You know we did aim to do 
it twice a week and I’m sure that | wasn’t on my own in losing that quite quickly. Just time constraints 
and having so much else to do on the curriculum—it really was impossible.’ 


For Year 6 classes, the understandable focus on SATs in the final year of the trial made it even more 
difficult to implement PATHS at anywhere near the recommended dosage: ‘I think it would be later on 
in the year... [when] SATs have gone, then that is probably when | will get time to do them’. This 
perceived lack of time also extended to being able engage with the recommended level of coaching 
support: 


‘She’s tried so hard to get in and help us with lessons and come in and observe us and teach us, and 
something else always just happens.’ ‘When she needs to talk to you about something and it’s very 
difficult because [of the] limited amount of time, even five minutes is a big chunk of your time at lunch 
time.’ 


Technical support and assistance 
The technical support and assistance (or ‘coaching’) aspect of PATHS implementation met with mixed 


reactions. Very few teachers had experienced what they perceived to be such an intensive model of 
external support. Nearly all respondents felt that coaching was important and valued its intended 
purpose—to maximise implementation effectiveness: ‘Oh you need it, definitely’. Despite this, for some 
it was viewed as unnecessary, either because they were confident in their own skills and knowledge 
(‘To be honest, | don't mean to undervalue anyone’s role, but | think the teachers who are implementing 
this PATHS course are well able to do it and know what's expected’), or because they viewed the 
coaching model as intrusive (‘! wouldn't want someone checking up on me all of the time’). Some felt 
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that continual support throughout the year was not needed (‘Once you're up and running with it... | 
doubt I'll need [coaching] as much for the rest of the year’). 


For others, however, the availability of external support was viewed positively: it helped to build 
confidence (‘She has come in and modelled a lesson for me, which was nice, and it was nice to know 
that | wasn't doing things wrong’); it provided valued opportunities for feedback and sharing of practice 
(‘It was good because she was feeding back about other schools’ experiences’); it engendered a sense 
of feeling collegiality and feeling supported (‘She’s always on email if need be, and she readily makes 
herself available to you and she emails a PATHS newsletter and things like that’); and in many cases 
served a motivational purpose (‘/t reinvigorates you’). 


School leadership 
The involvement of school leadership also emerged as a critical factor affecting implementation. Where 


school leaders were perceived to be supportive of PATHS, teachers felt better equipped to deliver it in 
class and viewed it as more likely to become embedded across the school: 


‘Our head asked me about some of the things from PATHS and he did an assembly linked with some 
of the problem-solving... it’s sort of dripping it into other parts of the school;’ ‘They support it to make 
sure it’s run consistently... throughout the school going to lunchtime organisers.’ 


Some headteachers took on an active monitoring role, which was viewed as increasing the likelinood 
of consistent delivery: 


‘Even though [the headteacher] isn’t administering it... having him as someone who Is saying “right I’m 
going to check out what is happening in PATHS”... | think knowing that he is there has been really, 
really useful. Because otherwise... it’s not coming down from the top... they need to put it on a higher 
profile for everyone to realise how important it is.’ 


However, in some cases, school leadership influence was viewed as a barrier to implementation. For 
example, one teacher reported having the class timetable set inflexibly by the headteacher (‘It’s not 
negotiable’), which left only a small, token amount of time for PATHS lessons (‘At the end of the day, 
which is not a good time... 3pm to 3.15pm, the children go home at 3.15pm’). This created a frustrating 
situation in which there was the will, but not the way, to deliver PATHS regularly: ‘Its easy to implement 
if you're given the time. If you’re not given the time it isn’t’. 
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Conclusion 


Key Conclusions 


1. Overall, PATHS did not have a positive impact upon children’s academic attainment. 


3. Higher levels of PATHS implementation quality and reach were associated with better 
academic outcomes 

4. Teachers reported a lack of time to implement PATHS at the recommended frequency 
On average, only half of the lessons were delivered. 


5. While PATHS does not appear to be a cost-effective way of improving students’ 
attainment, it may have an impact on other, non-cognitive outcomes. A full report on the 
NIHR trial looking at these outcomes will be published at a later date. 


2. The evidence does not indicate that PATHS produced differential gains for children 
eligible for free school meals (FSM). 


The primary aims of this study were: 


1. to assess the impact of PATHS on children’s academic attainment; 

2. to determine if PATHS produces ‘differential gains’ in academic attainment for children eligible 
for free school meals (FSM); 

3. to assess the extent to which PATHS implementation variability is associated with variability in 
children’s academic outcomes; and 

4. to identify process-related issues that might serve to illuminate the mechanisms underpinning 
findings for 1, 2 and 3 above. 


We utilised a cluster-randomised controlled trial design with a parallel implementation and process 
evaluation in order to achieve these aims. In relation to the first aim, our analyses did not produce any 
evidence of the impact of PATHS on children’s attainment in English (reading) or maths. Indeed, the 
only statistically significant finding at the intention to treat level favoured schools in the ‘usual practice’ 
arm of the trial (Year 6 English). At the sub-group level of analysis (the second aim above), there was 
no reliable evidence that PATHS produced differential gains in attainment for children eligible for free 
school meals. Although there was a marginal, non-significant trend indicating greater gains in English 
among FSM-eligible Year 6 pupils in PATHS schools, the associated effect size was extremely modest, 
being the equivalent of only two months’ progress compared to equivalent pupils in usual practice 
schools. 


Implementation analyses (the third aim above) revealed that PATHS implementation variability was 
associated with variability in children’s academic outcomes. Variability in usual SEL practice (targeted), 
quality, and reach each predicted child-level outcomes in at least 2 of our 4 analyses, with the largest 
ES being associated with increased implementation quality. Finally, our evaluation of process (the fourth 
aim) produced useful explanatory data. Of particular note in the light of the results outlined above was 
teachers’ perceived lack of time to implement PATHS at the recommended frequency: this was ascribed 
to a variety of issues including competing priorities. 


Limitations 


As with any study, this research was subject to limitations. First, the problems experienced with the 
InCAS measure (See Methods section) necessitated a design change—not ideal in the context of a 
major trial of this nature. However, the use of a post-test design (with prior attainment data used as a 
baseline covariate) is considered to be an extremely robust alternative, and ultimately our analyses 
proved to be adequately powered, with MDES ranging from 0.17 (Year 6) to 0.19 (Year 5). In the case 
of the Year 6 analyses we were also able to benefit from an unbiased sample as the use of attainment 
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data derived from the NPD meant that there was zero school attrition. Second, given that improvements 
in academic attainment are viewed as a distal outcome of PATHS, a case could be made that our trial 
design was not of sufficient length to allow measurable improvements in attainment to be triggered. 
However, this view is incongruent with the findings of the aforementioned PATHS trial by Schonfeld et 
al. (2014) that reported improvements in both reading and maths over an equivalent period of time 
(albeit the attainment of a basic level of proficiency as opposed to the degree of change in test scores 
reported in this study). Furthermore, the three recent meta-analyses of universal SEL interventions 
(Durlak et al., 2011; Sklad et al., 2012; Wigelsworth, Lendrum and Oldfield, in press) each reported 
meaningful improvements in academic attainment for interventions over much shorter periods of time 
(for example, 77% of the interventions reported by Durlak et al. (2011) lasted less than one year). 


A third and more plausible limitation relates to the low dosage rates identified in our IPE (as a reminder, 
our data indicated that teachers were implementing PATHS at an average of half the recommended 
frequency—one lesson per week). Although this is not a limitation of our research design per se, it could 
be argued that the apparent failure of PATHS to improve children’s academic attainment in this trial 
was attributable to implementation failure. To draw an analogy from pharmacology, for the intervention 
to trigger distal changes in outcomes such as attainment, there may be a ‘minimum effective dose’ 
(MED) (Liu, 2010) that PATHS schools failed to reach. The SEL theory of change outlined earlier would 
seem to support this—children presumably need a certain level of consistent exposure in order to 
produce the kind of meaningful changes in their social and emotional skills (Such as being better able 
to manage their behaviour, understand their emotions and work well with others) and improvements in 
their classroom climate that could then feasibly influence how well they do in school academically. Given 
the established dose-response relationships found elsewhere (for example Durlak and DuPre, 2008; 
Schonfeld et al., 2014) the failure of our implementation analyses to identify any relationship between 
dosage variability and outcome variability provides some confirmatory evidence for this hypothesis. As 
the MED was not reached overall, relative variability in dosage within the sample proved to be 
insignificant. However, even if this is indeed the case, questions are still raised in relation to the 
feasibility of PATHS as a tool for improving the attainment of children in England. That is, if schools in 
a major trial in which training, materials and external support and assistance were made available at no 
cost were not able to deliver PATHS at the frequency and consistency required to trigger academic 
change, what is the likelinood that schools will be able to do this in typical circumstances and 
conditions? 


A fourth set of limitations pertains to our assessment of implementation. Although generally considered 
to be a more valid means through which to assess implementation than the more frequently used 
teacher self-report surveys (Humphrey, 2013), independent observations only provide a snapshot of 
activity that may or may not be representative of implementation activity across the school year. 
Furthermore, it is impossible to rule out so-called ‘observer effects’ (that is, a change in behaviour as a 
result of knowing one is being observed). Finally, it is important to acknowledge that a number of the 
aspects of implementation that were assessed in this study could feasibly be viewed as a proxy for 
latent factors that co-vary with children’s attainment. So, for example, the quality of implementation of 
PATHS may simply be a proxy for teaching quality more generally. As we did not have the resources 
to observe teachers implementing non-PATHS lessons, this cannot be ruled out. Similarly, reach may 
be a proxy for attendance patterns within a given school. 


Interpretation and generalizability 


What is to be made of the findings of this study? It is important to consider the extent to which they may 
be considered robust and secure, and in light of this we refer to the EEF’s guidance on classifying the 
security of trial findings (see Appendix 4), which assigns up to 5 ‘padlocks’ for each of 5 key criteria, 
with a higher number of padlocks equating to greater security. 
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The Year 6 findings were awarded the full 5 padlocks, while the Year 5 findings were awarded 4 (due 
to attrition during the trial and the fact that the administrators of the Year 5 test were not blind to the 
treatment status of participants). Overall, the findings of this study can be considered to be robust and 
secure. Additionally, it is worthy of note that the study also complied with the EEF’s various quality 
markers. In terms of trial set-up, the study was registered at ISRCTN and a protocol was produced that 
was made available online: 


http://educationendowmentfoundation.org.uk/projects/promoting-alternaive-thinking-strategies-paths- 
manchester-university/ 


The outcome measures used were evaluated by the EEF and considered to be reliable, externally valid 
and non-intervention specific. Outcome tests were marked by individuals or organisations that were 
blind to each school’s allocation status. Randomisation was conducted independently of the evaluation 
team by a statistician in the MAHSC-CTU. All analysis and reporting is based on CONSORT standards 
and EEF guidelines. 


In terms of generalizability, while the robustness and security of our findings noted above are important, 
it is worth questioning the extent to which the composition of our study sample mirrors that of schools 
and children across England—are our schools and children ‘typical’? On this front we are in relatively 
good shape, albeit with a school sample that reflects the primarily urban setting of the trial schools 
(somewhat larger schools, with higher proportions of pupils eligible for FSM and speaking EAL than is 
the case nationally). Of note here is the fact that the trial schools were spread across seven diverse 
LAs. 


Noting the security of our findings and their likely generalizability, it is not possible to recommend 
PATHS as an effective intervention for improving the academic attainment of children in English primary 
schools. Of course, this conclusion comes with some caveats (see earlier), chief amongst which is the 
likely lack of an MED being reached, in other words, the findings may be due to implementation failure 
as opposed to programme/theory failure. We also note the marginal, non-significant trend toward higher 
English SAT scores for FSM-eligible pupils in PATHS schools when compared to their counterparts in 
usual practice schools. Given the low cost per pupil of the intervention, we can speculate that PATHS 
is, perhaps, an inexpensive vehicle to produce relatively small improvements in attainment for this 
specific sub-group, but further research would be needed to confirm this. Specifically, this study was 
initially powered for ITT rather than sub-group analysis, and we would need to determine a means 
through which to ensure that the MED is reached across schools/classes in order to produce practically 
meaningful effects. In relation to the latter, close attention would need to be paid to the factors affecting 
implementation that were identified through our process evaluation. 


Future research and publications 


As noted earlier, the research documented in this report is an augmentation of a larger trial funded by 
the NIHR. The main trial focused on the impact of PATHS on children’s social and emotional wellbeing, 
health-related quality of life, and other outcomes. The findings of this trial will be published and 
disseminated in due course, alongside papers derived from our IPE. We also anticipate publications 
based on the findings reported here to appear in academic and practitioner-oriented periodicals. The 
evaluation team is currently engaged in a follow-up phase, tracking the outcomes of the Year 6 cohort 
(who are now in secondary school) for a further two years in order to identify any sleeper or maintenance 
effects among our primary and secondary outcomes. 


In terms of future research on PATHS, we feel that further study is warranted regarding (a) the potential 
impact upon Year 6 English scores for FSM-eligible pupils, (b) how best to facilitate and document a 
MED for PATHS in relation to attainment, and (c) the possible interaction between (a) and (b), that is: 
if the conditions for a PATHS MED can be facilitated, does this produce more meaningful and 
measurable changes to Year 6 English scores among FSM-eligible pupils? 
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Appendix 1: Sample Parent information and consent sheet 


Education 
Endowment 
Foundation 


National Institute for 
Health Research 


PATHS TO SUCCESS 


INFORMATION SHEET FOR PARENTS 
Your child’s school is involved in an exciting project about the Promoting Alternative Thinking 
Strategies (PATHS) curriculum, called ‘PATHS To Success’. PATHS is a programme for all children 
that helps them to manage their behaviour, understand their emotions and work well with others. Our 
research project will help us to understand if PATHS works for children in Years 3-6. The project is 
funded by the National Institute for Health Research. 
We are writing to you because your child's school is involved in the project and we would like to know 
what you think about it. We will collect your views, those of your child, and his/her teacher once a 
year starting June/July 2012 (see below). 


Please take time to read the following information carefully and decide whether or not you would like 
to take part. 


If you would like any more information or have any questions about the research project, please 
telephone Dr Alexandra Barlow on 0161 275 3504 or email her at 
alexandra.barlow@manchester.ac.uk. 


Who will conduct the research? 


The research will be conducted by Professor Neil Humphrey and his research team at the School of 
Education, University of Manchester, Oxford Road, Manchester M13 9PL. 


Title of the research 
“PATHS To Success” 
What is the aim of the research? 


Our main aim is to examine the impact of the PATHS curriculum on the social and emotional 
wellbeing of children in primary schools in England. 


Where will the research be conducted? 
Primary schools in Greater Manchester. 
What is the duration of the research? 


The project itself runs from January 2012 until August 2017. The schools that implement PATHS 
(see below) will do so from September 2012 to July 2014. 


Why have | been chosen? 
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We are writing to you because your child’s school is taking part in the PATHS to Success Project. 
Schools will be randomly chosen to (a) implement PATHS over a two year period (PATHS schools), 
or (b) continue as normal (comparison schools). We will be collecting data in both PATHS and 
comparison schools. After two years, all schools will be free to decide whether they wish to 
start/continue using PATHS. 


What would | be asked to if | took part? 


You will be asked to complete a brief online survey about your child’s strengths and difficulties. Your 
child’s class teacher will complete a similar survey. 


Your child will be asked to complete a short survey about how well they work with others and how 
they feel about themselves and their school. If your child needs help to do this, they will be able to get 
support from a member or school staff or one of our researchers. 


These surveys will be completed three times — in June/July 2012, 2013 and 2014. They will take 
approximately 10 minutes to complete each time. 


In consenting to take part you are also giving permission for your child and his/her teacher to 
complete these surveys. 


If you do not have access to the internet we will be happy to either provide a paper copy or complete 
it over the telephone with you at an agreed time. If you would like to do this please contact Dr 
Alexandra Barlow (details below) and she will arrange this for you. 


What happens to the data collected? 


The data will be analysed by our research team at the University of Manchester. We will write a report 
based on our analyses for the National Institute for Health Research. It is also likely that we will write 
articles for academic journals based on what we find out in the project. Finally, it is possible that we 
will write a book about the research. Your child’s name will not be used in any of the reports that we 
write. 


How is confidentiality maintained? 


All data provided will be treated as confidential and will be completely anonymous. Identifying 
information ( your child’s name) will only be used in order to match responses about the same 
individual from different respondents (e.g. parents and teachers) and across different times (e.g. 
June/July 2012, 2013, and 2014). After this matching process is complete, all identifying information 
will be destroyed. 


The website that houses the survey will be completely secure and password protected. All survey 
data will be stored on a secure, password protected computer to which only senior members of the 
research team have access. 

What happens if | do not want to take part or | change my mind? 


It is up to you if you want to take part. 


If you decide to take part you do not need to do anything — you will be sent further details about when 
and how to complete the survey in the near future. 


If you decide not to take part then you need to either complete the opt-out consent form enclosed and 
return it to our research team at the address above or contact Dr. Alexandra Barlow by telephone or 
email (details above) by Friday 1st June 2012. 


If you decide to take part and then change your mind, you are free to withdraw at any time without 


needing to give a reason. If you do this please rest assured that we will destroy any data collected 
about your child as part of the study. 
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Will | be paid for participating in the research? 
We are not able to offer any payment or incentive for participating in this study. 
Criminal Records Check 


Every member of our research team has undergone a Criminal Records Bureau check at the 
Enhanced Disclosure level. 


Contact for further information 


Dr. Alexandra Barlow 

Educational Support and Inclusion 

School of Education 

University of Manchester 

Oxford Road 

Manchester 

M13 9PL 

Tel: 0161 275 3504 

Email: alexandra.barlow@manchester.ac.uk 


Also, please see our website for further details about the PATHS curriculum and background, the 
project design and project team. 


The website can be found at: www.pathstosuccess. info 

What if something goes wrong? 

If completing the survey makes you worry about your child’s wellbeing then you should contact the 
school in the first instance and ask to speak to his/her teacher. 

You can also get independent support and advice from a charity called Young Minds. Their parent 
helpline number is 0808 802 5544. 

If you ever wish to make a formal complaint about the conduct of the research you should contact the 


Head of the Research Office, Christie Building, University of Manchester, Oxford Road, Manchester 
M13 9PL. 
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é Education 
Endowment 
Foundation 
National Institute for 
Health Research 
PATHS TO SUCCESS 
PARENT CONSENT FORM 


An information sheet is attached to this form. Please read it carefully before making a decision about 
taking part. 


If you are willing to take part then you do not need to do anything at the moment. 


If you decide not to take part, then you need to complete the opt-out consent form below and use the 
freepost code below and return it to: 


FREEPOST RLYU-KAAB-AXRC 
Dr Alexandra Barlow, 

Educational Support and Inclusion, 
School of Education, 

University of Manchester, 

Oxford Road, 

Manchester, 

M13 9PL. 


Alternatively, Dr. Barlow can be contacted by telephone on 0161 275 3504 or email at 
alexandra.barlow@manchester.ac.uk. If you do not wish to participate please let us know by Friday 
1st June 2012. 


Finally, please also remember that if you do decide to take part, you are free to change your mind at 
any point in the study. 


| do not wish to participate in the PATHS to Success project. My details are as follows: 
My name 
My child’s name 
Name of my child’s school 


Signed: Date: 
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Appendix 2: Impact analysis multi-level model tables 

NB: All continuous data in Tables 6-13 was standardised prior to analysis to facilitate comparison of 
ES within and across models (all ES were then converted to Hedge’s g as per EEF specifications—see 
right hand column). 


Table 6: Year 5 Reading 


Empty model BX) miraveye(=)| 
Boi = —0.049 (0.051) Boi = 0.184 (0.079) 
Level Co-efficient SE p Co- SE p 
B efficient B 
School 0.091 0.024 .001 0.031 0.011 | <.001 
(9%) (3.5%) 
FSM -0.007 0.002 | <.001 
EAL -0.000 0.001 ns 
If PATHS 0.022 0.078 ns 
Pupil 0.638 0.030 | <.001 0.574 0.028 | <.001 
(63.2%) (65%) 
If FSM -0.546 0.078 | <.001 
If female 0.320 0.043 | <.001 
Time 0.280 0.012 | <.001 0.278 0.012 | <.001 
(27.8%) (31.5%) 
If Post-test -0.032 0.038 ns 
Interactio Group*time -0.029 0.052 ns -0.029 
n Group*FSM 0.087 0.106 ns 
FSM*time 0.006 0.077 ns 
Group*FSM*time 0.017 0.102 ns 0.017 
-2*Loglikelihood 7007.149 6692.027 
X2 (27, n = 2730) = 315.122, p < .001 


Table 7: Year 5 Maths 


Empty model Test model 
Boi = —0.053 (0.058) Boi = 0.352 (0.099) 
Co-efficient B Co-efficient B SE 

School 0.127 0.03 | <.001 0.063 0.018 | ns 
(12.8%) 2 (7.1%) 

FSM -0.007 0.002 | <.0 

01 

EAL -0.003 0.002 ns 

If PATHS 0.013 0.095 ns 

Pupil 0.856 0.02 | <.001 0.539 0.027 | <.0 

(57.5%) 8 (60.4%) 01 

If FSM -0.458 0.078 | <.0 

01 

If female 0.045 0.043 ns 

Time 0.293 0.01 | <.001 0.291 0.013 ns 
(29.7%) 2 (32.6%) 

If Post-test -0.020 0.039 | ns 

Interaction Group*time 0.026 0.053 sons 0.026 
Group*FSM 0.158 0.107 | ns 
FSM*time -0.060 0.079 | ns 
Group*FSM‘*time -0.036 0.104 ns -0.036 
-2*Loglikelihood 6906.135 6664.674 
X? (27, n = 2712) = 241.461, p < .001 
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Tia} e)avarareye(=1| Test model 
Boi = — 0.032 (0.052) Boi = 0.110 (0.084) 
Level Co-efficient SE p Co-efficient SE p 
B B 
School 0.094 0.025 | <.001 0.036 0.013 | .004 
(9.4%) (4%) 
FSM -0.008 0.002 | <.001 
EAL -0.001 0.002 ns 
If PATHS 0.201 0.083 | .011 
Pupil 0.666 0.029 | <.001 0.618 0.027 | <.001 
(66.3%) (69.1%) 
lf FSM -0.362 0.084 | <.001 
If female 0.266 0.044 | <.001 
Time 0.244 0.009 | <.001 0.240 0.009 | <.001 
(24.3%) (26.9%) 
If post-test 0.063 0.030 | .018 
Interaction Group*time -0.106 0.043 | .007 -0.106 
Group*FSM -0.066 0.111 ns 
FSM*time 0.072 0.058 ns 
Group*FSM*time 0.120 0.078 | .062 0.120 
-2*Loglikelihood 7471.066 7157.864 
X? (28, n = 3062) = 313.202, p< .001 


Table 9: Year 6 Maths 


Empty model Test model Hedges g 
Boi = —0.033 (0.051) Boi = 0.300 (0. 086) 
Level Co-efficientB = SE p Co-efficient B SE p 
School 0.094 0.02  <.001 0.038 0.013 | .003 
(9.4%) 5 (4.1%) 
FSM -0.007 0.002  .001 
EAL -0.001 0.002 | ns 
If PATHS 0.158 0.085 | .036 
Pupil 0.606 0.02 | <.001 0.579 0.027 | <.0 
(60.7%) 8 (63.2%) 01 
If FSM -0.404 0.085 | <.0 
01 
If female -0.073 0.044  .049 
Time 0.299 (29.9%) | 0.01 | <.001 0.299 0.011 | <.0 
1 (32.6%) 01 
If post-test 0.003 0.034 | <.0 
01 
Interaction Group*time -0.025 0.047 ns -0.025 
Group*FSM 0.035 0.113 ns 
FSM*time 0.017 0.064 ns 
Group*FSM*time 0.016 0.087 ns 0.016 
-2*Loglikelihood 7722.919 7472.368 
X? (28, n = 3062) = 250.551, p < .001 
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Appendix 3: Implementation analysis multi-level model 
tables 


Table 10: Year 5 Reading 


Empty model BX) mi raveye(:)| 
Boi = —0.045 (0.066) Boi = 2.90 (0.736) 
Level | Co-efficient SE p Co-efficient B SE p 
B 

School 0.048 0.034 ns School 0.000 0.000 ns 

FSM -0.001 0.003 ns 

EAL 0.002 0.002 ns 

Universal SEL -0.013 0.017 ns 

Targeted SEL 0.001 0.022 ns 

Class 0.045 0.031 ns Class 0.000 0.000 ns 

Dosage (compared to -0.032 (if mod) 0.102 ns 

low) 0.000 (if high) 0.000 ns 

Fidelity (compared to 0.032 (if mod) 0.164 ns 

low) -0.316 (if high) 0.319 ns 

Quality (compared to 0.109 (if mod) 0.263 ns 

low) 0.315 (if high) 0.351 ns 

Responsiveness 0.180 (if mod) 0.299 ns 

(compared to low) 0.000 (if high) 0.000 ns 
Reach (compared to 0.323 (if mod) 0.146 | .019 

low) -0.178 (if high) 0.165 ns 
Observed week -0.144 0.035 | <.001 

number 
Pupil 0.662 0.043 | <.001 | Pupil 0.603 0.048 | <.001 
If female 0.351 0.077 | <.001 
If FSM -0.433 0.090 | <.001 
Time 0.289 0.016 | <.001 | Time 0.278 0.019 | <.001 
If post-test -0.087 0.035 | .006 
-2*Loglikelihood = 3729.046 -2*Loglikelihood = 2324.768 
X? (25, n = 958) = 1404.278, p < .001 
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Empty model Test model 
Boi; = —0.064 (0.071) Boi = 3.479 (0.922) 
Co-efficient Co-efficient B 
B 
School 0.052 0.039 | ns_ | School 0.000 0.000 ns 
FSM -0.001 0.004 ns 
EAL -0.001 0.002 ns 
Universal SEL -0.017 0.019 ns 
Targeted SEL 0.012 0.025 ns 
Class 0.073 0.037 | .029 | Class 0.016 0.013 ns 
Dosage (compared to -0.050 (if mod) 0.146 ns 
low) 0.000 (if high) 0.000 ns 
Fidelity (compared to 0.021 (if mod) 0.184 ns 
low) -0.140 (if high) 0.353 ns 
Quality (compared to 0.065 (if mod) 0.300 ns 
low) 0.066 (if high) 0.390 ns 
Responsiveness 0.301 (if mod) 0.337 ns 
(compared to low) 0.000 (if high) 0.000 ns 
Reach (compared to 0.341 (if mod) 0.173 =.030 
low) -0.212 (if high) 0.189 ns 
Observed week -0.166 0.029 <.001 
number 
Pupil 0.512 0.037 | <.00 Pupil 0.468 0.044 <.001 
1 If female 0.110 0.070 ns 
lf FSM -0.347 0.082 <.001 
Time 0.317 0.018 | <.00 Time 0.283 0.019 <.001 
4 If post-test -0.066 0.036 | .036 


-2*Loglikelihood = 3615.510 


-2*Loglikelihood = 2221.771 


X? (25, n = 948) = 1393.739, p < .001 
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Table 12: Year 6 English 


Empty model Test model 
Boy = 9.024 (0.074) Boy = —1.271 (0.741) 
Level Co-efficient Co-efficient B 
B 

School 0.055 0.042 ns School 0.000 0.000 ns 
FSM -0.009 0.004 | .019 

EAL 0.003 0.003 ns 
Universal SEL 0.031 0.019 | .060 

Targeted SEL -0.014 0.023 ns 
Class 0.077 0.039 | .029 Class 0.029 0.016 | .044 

Dosage (compared -0.001 (if mod) 0.115 ns 

to low) 0.000 (if high) 0.000 ns 

Fidelity (compared -0.177 (if mod) 0.168 ns 

to low) 0.031 (if high) 0.259 ns 
Quality (compared 0.691 (if mod) 0.302 | .017 
to low) 0.622 (if high) 0.381 | .060 

Responsiveness -0.345 (if mod) 0.256 ns 

(compared to low) -0.123 (if high) 0.304 ns 
Reach (compared to 0.458 (if mod) 0.170 | .007 

low) 0.151 (if high) 0.152 ns 
Observed week 0.063 0.036 | .049 

number 

Pupil 0.575 0.037 <.001 Pupil 0.548 0.039 | <.001 
If female 0.324 0.065 | <.001 
If FSM -0.398 0.079 | <.001 
Time 0.263 0.013 <.001 Time 0.255 0.014 | <.001 

If post-test -0.010 0.028 ns 

-2*Loglikelihood = 3872.527 -2*Loglikelihood = 3038.233 


X2 (25, n = 1308) =834.294, p < .001 
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Table 13: Year 6 Maths 
Empty model Test model 


Boy = 9.031 (0.069) Boy = —9.995 (0.679) 
Level | Co-efficient B Co-efficient B 
School 0.048 0.0 ns School 0.041 0.021 | .033 
37 FSM -0.012 0.005 | .014 
EAL 0.004 0.003 ns 
Universal SEL 0.025 0.023 ns 
Targeted SEL 0.004 0.028 ns 
Class 0.062 0.0 | .044 Class 0.000 0.000 ns 
34 Dosage (compared -0.069 (if mod) 0.121 ns 
to low) 0.000 (if high) 0.000 ns 
Fidelity (compared -0.104 (if mod) 0.191 ns 
to low) -0.036 (if high) 0.257 ns 
Quality (compared 0.682 (if mod) 0.256 | .008 
to low) 0.695 (if high) 0.343 ns 
Responsiveness -0.422 (if mod) 0.225 | .038 
(compared to low) -0.261 (if high) 0.290 ns 
Reach (compared to 0.298 (if mod) 0.164 | .043 
low) 0.190 (if high) 0.166 ns 
Observed week 0.061 0.035 | .049 
number 
Pupil 0.555 0.0 | <.001 Pupil 0.562 0.042 | <.001 
37 If female -0.013 0.068 ns 
If FSM -0.382 0.082 | <.001 
Time 0.313 0.0 | <.001 Time 0.326 0.018 | <.001 
16 If post-test 0.005 0.032 ns 
-2*Loglikelihood = 4010.200 -2*Loglikelihood = 3243.888 
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Appendix 4: Security classification of trial findings 


2. Power: 5. Threats to 
1.Design: lwhaté 3. Attrition: 4. Balance: F validity ae 
What is the What is the Adjustment aoe 
; minimum Adjustment to 
quality of the (AV7-) Me) ite | ge) to rating to 
fo [14-104 €-] 6) (= account for issues 
design of the out from the account for 
evaluation? lsh evaluation? balance? wile 
start? ; : interpretation? 
Evaluation design Implementation Analysis and interpretation 


aeeLihare| Pn PX=X-Fe fa) 3. Attrition 4. Balance 5. Threats to 
validity 


Fair and clear experimental 
design (RCT, RDD) < 0.3 


Well-matched comparison 
(quasi-experiment) < 0.4 < 30% 
Matched comparison 
(quasi-experiment) <0.5 < 40% 
oor or no matchin <0. 
Pp g y y 
No comparator Imbalanced on en 


Padlock factors shown in light green for Year 6 results, and dark green for year 5 results. 


The final security rating for this trial is 4 @ for the Year 5 results, and 5 @ for the year 6 results. This 
means that the Year 5 conclusions have moderate security, and the Year 6 results have high security. 


This evaluation was designed as a randomised controlled trail. The sample size was designed to 
detect a MDES of less than 0.2 for both year groups. Only 37 of the 45 schools (82%) randomised 
were followed up for Year 5 testing, reducing the security for this year group to 4 @. All schools were 
followed up for Year 6 tests. Balance at baseline was high. The post-tests in year 5 were administered 
by Research Assistants that were not blind to the treatment status of participants, incurring a potential 
threat to the validity of the findings for this year group, but not enough to reduce the security to 3 @. 
Therefore, the final security rating is 4 @ for Year 5 and 5@ for year 6. 
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Appendix 5: Cost rating 


Cost rating Description 


£ Very low: less than £80 per pupil per year. 
£E Low: up to about £200 per pupil per year. 


LEE Moderate: up to about £700 per pupil per 
year. 


EEEE High: up to £1,200 per pupil per year. 


EEEEE Very high: over £1,200 per pupil per year. 
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